Skip to content

refactor(engine): manage child process lifecycles#2160

Merged
jrusso1020 merged 3 commits into
mainfrom
07-10-refactor_engine_manage_child_processes
Jul 17, 2026
Merged

refactor(engine): manage child process lifecycles#2160
jrusso1020 merged 3 commits into
mainfrom
07-10-refactor_engine_manage_child_processes

Conversation

@jrusso1020

@jrusso1020 jrusso1020 commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

What

Give FFmpeg and related subprocesses one managed lifecycle.

Why

Timeout, cancellation, stderr bounds, and SIGTERM/SIGKILL behavior were duplicated and inconsistent across encoding phases.

How

Add ManagedChildProcess with composed signals, grace escalation, guaranteed reap, and typed termination reasons, then migrate the targeted encode/probe/extract paths in this stack slice.

Test plan

  • managed process, encoder, ffprobe, extraction, and assembly tests
  • Stack-wide lint, format, build, typecheck, and relevant integration gates

jrusso1020 commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from 5e82fec to fc61671 Compare July 13, 2026 23:08
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_add_browser_leases branch from 82cdfe7 to 5f2aacb Compare July 13, 2026 23:08
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch 2 times, most recently from b683ddb to 4e86382 Compare July 14, 2026 03:20
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_add_browser_leases branch 2 times, most recently from 78ee37c to 263a1a1 Compare July 14, 2026 17:03
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch 2 times, most recently from 1a7d917 to 7dd4686 Compare July 14, 2026 17:11
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_add_browser_leases branch from 406ba14 to a53a699 Compare July 14, 2026 17:35
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from 7dd4686 to 8733675 Compare July 14, 2026 17:35
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_add_browser_leases branch from a53a699 to ae6769c Compare July 14, 2026 17:57
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from 8733675 to 223e861 Compare July 14, 2026 17:57
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_add_browser_leases branch from ae6769c to 3335e20 Compare July 14, 2026 18:06
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch 4 times, most recently from eda0bb1 to c0bd5b9 Compare July 14, 2026 19:47
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_add_browser_leases branch from 248ec97 to d35fc32 Compare July 14, 2026 19:55
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch 2 times, most recently from 571341a to 606e05b Compare July 14, 2026 23:29
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_add_browser_leases branch from d35fc32 to 4f12432 Compare July 14, 2026 23:29

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The centralization is valuable, and the migrated callers consistently propagate typed termination reasons and bounded stderr. I found one lifecycle bug in the new primitive that breaks the guarantee this PR is built around.

[P1] Do not treat every ChildProcess error as a spawn failurepackages/engine/src/utils/managedChildProcess.ts:102

Node emits ChildProcess error not only when spawn fails, but also when a running process cannot be killed (and for failed IPC sends / spawn-signal aborts). Today any such error immediately calls settle("spawn_error"). settle() then clears the SIGKILL escalation timer and removes the close listener (:159-176), while processTracker also drops the child on error (packages/engine/src/utils/processTracker.ts:5-10).

That means this concrete sequence leaks the process and lets the caller continue before reaping it:

  1. deadline/abort calls kill("SIGTERM") and schedules SIGKILL (:134-156);
  2. signal delivery fails and the child emits error while it can still be alive;
  3. onError resolves wait() as spawn_error, cancels escalation, and removes the only close/reap listener.

The same premature settlement can occur for any post-spawn error, and kill() returning false is currently ignored. Please distinguish a true pre-spawn failure from runtime/termination errors. A post-spawn kill error must preserve the requested termination reason and the escalation/reap path (or otherwise synchronously prove the child is gone). Add a fake-child regression that aborts, emits error after SIGTERM, verifies wait() remains pending, observes SIGKILL after the grace period, and settles only on close.

This is directly documented by Node: the error event includes “process could not be killed,” and kill() returning true only means the signal was delivered—not that the process terminated.

Non-blocking scope note: packages/producer/src/services/audioExtractor.ts:84-105 remains a raw, unbounded FFmpeg lifecycle despite the PR body saying encode/probe/extract paths are migrated. If that path is intentionally outside this slice, narrowing the PR wording would avoid overstating coverage; otherwise it should use the shared primitive in a follow-up.

Verdict: REQUEST CHANGES

Reasoning: Exact head 1608c0e0cdaafa1763bebc7599dd1ffcf1f8fe16 is current and all required CI is green, but the new shared lifecycle can resolve and cancel escalation on a post-spawn kill error while the child remains alive. That violates the core cancellation/reap contract and affects every migrated caller.

— Magi

@jrusso1020

Copy link
Copy Markdown
Collaborator Author

Addressed the lifecycle blocker on exact head 4326ed67f783b88ad1c3b20693a60b3bb5fe9010.

  • ManagedChildProcess now distinguishes true pre-spawn failures from post-spawn runtime/kill errors. Runtime errors no longer settle wait(), cancel escalation, or remove the close listener; the error listener remains active until reaping.
  • processTracker now retains children until close, rather than dropping them on a post-spawn error.
  • Added fake-child coverage for abort → SIGTERM → post-spawn error → SIGKILL → close, including a second failed kill event, plus tracker coverage.
  • Narrowed the PR wording to the targeted encode/probe/extract paths in this stack slice; audioExtractor.ts remains outside this PR.

Validation: focused lifecycle/tracker tests 12/12; full engine 1,101 passed / 3 skipped; engine typecheck; repository lint, format, tracked-artifact and pre-commit gates; full workspace build on exact restacked top 6be9c1514993bdcc045a1ddf319765c90454c36e. #2160#2174 are republished and fresh exact-head CI is running. I’ll request fresh approval once all required gates, including aggregate regression and JavaScript CodeQL, are green.

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Audited: all 17 changed files in the complete main...a167618d2 diff, with the lifecycle primitive, tracker, every migrated caller, and tests read end-to-end. Trusting: no generated artifacts (none changed).

The original blocker is fixed correctly in packages/engine/src/utils/managedChildProcess.ts:94-118: a true pre-spawn failure settles as spawn_error, while a post-spawn runtime/kill error leaves escalation and the close-based reaping path active. packages/engine/src/utils/managedChildProcess.test.ts:61-91 pins the requested SIGTERM → error → SIGKILL → close sequence.

[P1] Remove tracked children on exit as well as closepackages/engine/src/utils/processTracker.ts:5-9

Changing the tracker from exit to only close fixes the post-spawn-error drop, but creates a different dangerous window. Node’s exit event means the child process has ended; close waits for stdio to close and can be delayed when another process still holds those streams. During that gap the dead child remains in tracked with proc.killed === false, so killTrackedProcesses() calls proc.kill("SIGTERM") (:15-24) and later SIGKILL. Node explicitly warns that signaling a child after it exited can target an unrelated process if the PID has already been reassigned.

The current tracker tests do not cover this: the test named “removes it after exit” actually waits for close (packages/engine/src/utils/processTracker.test.ts:11-18). Preserve the post-spawn-error fix without retaining exited PIDs by removing on both terminal events:

proc.once("exit", remove);
proc.once("close", remove); // spawn failures may have no exit event

Keep error non-terminal, and add a regression that emits exit without close, calls killTrackedProcesses(), and verifies kill is not invoked.

Exact head a167618d2eb4c8822ed8592a943c1d8901ff10d7 is mergeable. All completed required checks are green; six regression shards are still pending, with zero failures. This finding is independent of those pending jobs.

Verdict: REQUEST CHANGES

Reasoning: The managed-process fix now preserves escalation and reaping after runtime errors, but the tracker’s close-only cleanup can signal a stale/reused PID after the child has already exited. Removing on both exit and close closes that safety gap without reintroducing the original error-event bug.

— Magi

@jrusso1020

Copy link
Copy Markdown
Collaborator Author

Addressed the follow-on processTracker P1 on exact head 018581b.

  • trackChildProcess now removes children on both exit and close, while error remains non-terminal.
  • Added a regression that emits exit without close, calls killTrackedProcesses(), and verifies kill() is not invoked.
  • Validation: processTracker 8/8; full engine 1,102 passed / 3 skipped; engine typecheck; lint and format; tracked-artifact and pre-commit gates; full workspace build on exact restacked top 356fdf7.
  • PRs refactor(engine): manage child process lifecycles #2160refactor(repo): resolve changed-code audit #2174 were restacked and republished; fresh exact-head CI is running.

I will request fresh approval once all required gates, including aggregate regression and JavaScript CodeQL, are green.

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Audited: the complete 17-file PR diff in the prior rounds, all existing review/comment history, and the exact R3 delta at 018581b22c06d83aa94866dd0c1135a87b4898f3. Trusting: no unverified generated artifacts.

The original lifecycle blocker remains fixed: ManagedChildProcess distinguishes spawn failure from post-spawn runtime/kill errors, so those later error events do not settle wait(), cancel SIGKILL escalation, or detach the close-based reaping path. The fake-child lifecycle regression still pins abort → SIGTERM → post-spawn error → SIGKILL → close.

The follow-on tracker blocker is also resolved at the correct ownership boundary:

  • packages/engine/src/utils/processTracker.ts:5-10 removes the tracked child on both exit and close, while leaving error non-terminal. This prevents shutdown from signaling a process that has exited but whose stdio has not closed, without regressing the reaping semantics.
  • packages/engine/src/utils/processTracker.test.ts:21-37 reproduces that exact exit-without-close interval and verifies shutdown sends no signal to the exited child.
  • The existing post-spawn-error coverage still verifies that an error alone does not remove a live child.

I found no remaining code findings. The exact head is current, open, and mergeable. Exact-head CI is still running with no failures; per James's request, this is the code verdict and does not treat pending CI as a failure.

Verdict: APPROVE
Reasoning: Both lifecycle ownership defects are fixed at their shared primitives and pinned by regressions that exercise the precise event-ordering failures; the R3 change introduces no new correctness issue.

— Magi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants